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ABSTRACT 

National- and state-level reports on education 
indicators have been produced since the mid-1980s. However, the 
debate continues over which education indicators to select and 
report. This paper outlines the lessons gained by a national 
organization in building an education indicators system through 
cooperation with state education agencies and federal agencies. Three 
issues in the development process for education indicators are 
emphasized: (1) the role of the consensus process among educators, 
researchers, and pol i C5nnakers in selecting and defining indicators; 
(2) hovT education indicators can be produced through a cooperative 
data system with states; and (3) methods for reporting indicators 
that increase their usefulness by policymakers and educators. The 
experiences of the Council of Chief State School Officers illustrate 
the variety of issues that need to be resolved, and the need for 
careful definition of the indicator areas and the data required. It 
is essential to involve different interests or perspectives on the 
indicators system in the development process. Two figures and two 
tables present information about the process. (Contains 33 
references . ) (Author /SLD) 
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Abstract 



National and state-level reports on education indicators have been produced since 
the mid-1980's. However, the debate continues over which education indicators to select 
and report. This paper outlines the lessons gained by a national organization in building 
an education indicators system through cooperation with state education agencies and 
federal agencies. Three issues in the development process for education indicators are 
emphasized: • 

1. The role of a consensus process among educators, researchers, and 
policymakers in selecting and defining indicators; 

2. How education indicators can be produced through a cooperative data system 
with states; and 

3. Methods for reporting indicators that increase their usefulness by 
policymakers and educators. 
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DE\nELOPING A SYSTEM OF EDUCATION INDICATORS 
Selecting, Implementing, and Reporting Indicators 

Education policymakers at national, state, and local levels wish to support efforts to 
develop education indicators that would provide a rehable, periodic snapshot of the 
condition of education in our schools and could be used to provide a valid basis for 
assessing educational improvement. National and state-level reports on education indicators 
have been produced since the mid-l980's. However, the debate continues over which 
education indicators to select and report. 

In 1991, a national panel authorized by Congress recommended that new categories 
and types of education indicators be developed to replace those currently used. Tne 
National Education Goals Panel recommended, and reponed, indicators for the six national 
goals but also identified the need for a variety of new indicators at national and state levels. 
The National Coimcil on Education Standards and Testing has recommended that new 
forms of student assessment be developed to determine progress toward national standards 
for student learning in core academic subjects. 

In the recent deliberations about education indicators, emphasis has been placed on 
determining which indicators should be available and what methods should be used to 
collect data to measure student, teacher, or school performance. However, the recent 
attention to indicators includes little discussion of the process for selecting indicators, such 
as whose interests and needs are represented, and far too little attention to how to obtain 
cooperation among national, state, and local agencies. 

This paper outlines the lessons gained by a national organization in building an 

Funding support for this paper was through a grant from the National Science Foundation, 
Education and Human Resources, Division of Research, Evaluation, and Dissemination. 



education indicators system through cooperation with state education agencies and federal 
agencies. The goal of the paper is to contribute to the understanding of education 
polic)'n:iakers, educators, and researchers about how education indicators can be effectively 
developed and used. Tnree issues are analyzed: 

1. The role of a consensus process among educators, researchers, and 
policj'makers in selecting and defining indicators; 

2. How education indicators can be produced through a cooperative data system 
with states; and 

3. Methods for reporting indicators that increase their usefulness by 
policymakers and educators. 

EDUCATION INDICATORS IN THE 1980's and '90's 

Education statistics have been reported in the U.S. since the 19th centur)'. Education 
indicators have become a major issue at national, state, and local levels of education over 
the past decade. Education "indicators" are selected statistics that are intended to inform 
policymakers, educators, and the public about the condition of the education system (Oakes, 
1986). The indicators focus on key aspects of how the system is currently functioning, 
whether progress is being made, and where there ?jt problems (Shavelson, McDonnell, 
Oakes, & Carey, 1987; Horn and Winter, 1989). Education indicators are selected to 
include statistics that have specific relevance to policies that direct and shape education. 

Two trends since the mid I980's gave rise to the current interest in education 
indicators: (a) the education reform movement aimed at raising the qualit)' of education 
in our schools, and (b) a renewed emphasis on accountabilit}' in education. Many state 
policy imtiatives in the 1980's aimed at education reform were stimulated by a series of 



national commissions and reports critical of the quality of education in our schools. Two 
of the reports- A Nation at Risk (National Commission on Excellence in Education, 1983) 
and Educating Americans for the 21st Century (National Science Board Comuriission on 
Science, Mathematics, and Technology Education, 1983)-outlined needs and strategies for 
education reform and also highlighted the need for greatly improved capacity for data to 
assess the quality of education and track the rate of progress. Much of the recent work on 
education indicators can be traced to these reports. 

Following A Nation at Risk , the Department of Education initiated an annual 
Condition of Education report, which ^'ncludes national statistics drawn largely from existing 
departmental surveys (National Center for Education Statistics, 1991). The 1987 Hawkins- 
Stafford amendments reauthorizing many federal education prograuis, mandated a national 
panel to examine the need for national education indicators. The panel's report. Education 
Counts , outlines a comprehensive model that includes a broad array of indicators and data 
that go far beyond the current surveys and data collection of the Department of Education 
(National Study Panel on Education Indicators, 1991). The Hawkins-Stafford amendments 
also authorized a Trial State Assessment of the National Assessment of Educational 
Progress (NAEP) which, if fully implemented, would greatly expand the scope and role of 
NAEP in pro\dding education indicators at state and national levels. 

Tht National Science Foundation (NSF) has also given impetus to the development 
and use of education indicators. Since 1985, the biennial Science Indicatorf; report has 
expanded the chapter on indicators of the qualit}' of elementary and secondary education 
(National Science Board, 1991). NSF supported major studies to assess the quality of 
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current data on science and math and to develop models for impro\dng science and 
mathematics indicators (Raizen and Jones, 1985; Shavelson, et al, 1987; Murnane and 
Raizen, 1988). 

The interest and support of state policymakers for education indicators and 
accountabiiit)' increased significantly with the education policy initiatives in the I980's that 
were aimed at reforming the quality of education.. The changes in state policies also 
accelerated the already increasing state role in education funding (Doyle and Hartle, 1985). 
By the end of the 1980's, a majority of states had established a system of indicators that 
were linked to the state role in education accountability (OERI, 1988). Education 
accountability was increasingly defined by states as accountability for outcomes (Malen and 
Fuhrman, 1991), as exemplified by the expansion of statewide student assessment programs 
to over 40 states (Blank and Schilder, 1991). 

Tne National Education Goals of the President and Governors produced additional 
\isibility and importance for national and state level indicators of education progress. The 
Goals Panel's first report outlined the desired indicators for each goal, and it showed that 
valid state-by-state data axe not currently available to measure progress toward several of 
the goals (1991). Tht report has led to extensive debate about the next steps in establishing 
standards for, and methods of assessing, student and school progress (National Council on 
Education Standards and Testing, 1992). 



STATE NETWORX FOR EDUCATION INDICATORS 

In 1985, the Council of Chief Stale School Officers changed its polic)' of opposing 



siaie-level education comparisons to approNing a plan for a system of state-by-state 
education indicators. The Council recognized the need for reliable, valid indicators of the 
condition of education in our schools and took the lead in efforts to develop high qualit)', 
comparable indicators, .such as the expansion of the National Assessment of Educational 
Progress to state level reporting. The model for state indicators approved by the Council 
includes three components: (a) student outcomes, (b) education policies and practices, and 
(c) state context (CCSSO, 1985, 1990). 

Tlie Council received a grant from the National Science Foundation to develop and 
repon state-level and national indicators of science and mathematics education. The project 
followed the Council's broader goal of working with state depanments of education to 
produce valid, comparable education indicators. Tne project was a part of NSF's goal of 
having reliable, periodic indicators of the quality of science and mathematics education at 
elementary and secondary levels. The Council and NSF staff anticipated that a cooperative 
state and national effort could produce better information about science and mathematics 
education that would be very useful to national and state education policymakers. The steps 
taken in the Council/NSF project in developing and reporting state science and mathematics 
indicators are likely to be applicable to the development of indicators in a wide range of 
education areas. 



STEPS IN DEVELOPING AN INDICATORS SYSTEM 

The process of developing an indicators system can be outlined in nine steps. 
Particular emphasis is given to those elements of the process can be used for planning and 
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implementing systems of education indicators at national, state, or local levels. The steps 
are organized under three major categories of activity': selecting indicators; organizing a 
cooperative system for data; and reporting comparative data on indicators. 
Selecting Indicators 

L Develop a Conceptual Framework Based on Research Results and Interests of 
Polic^TTiakers and Educators, 

Tne process of selecting state indicators for science and mathematics education began 
with a paper outlining a conceptual framework (Blank, 1986). The elements of the 
framework reflected both the results of research on the educational process in science and 
mathematics and the needs of policymakers and educators for valid, useful data. The 
interests of state policjmakers in education indicators were initially identified from the 
Council's model for state indicators (CCSSO, 1985), and the report of the National 
Governors Association, Time for Results (1986), which outlined a series of needed state-^ 
level education indicators. 

Recent research on the need for education indicators contributed to the conceptual 
frameworL A study by the Rand Corporation recommended that indicators be based on 
a model of the education system and outlined different approaches for developing indicators, 
such as a "patchwork approach" that would combine indicators from different sources, 
including national and state surveys (Shavelson, el al, 1987). An expert panel of the 
National Research Council recommended thai education indicators focus on elements of the 
education system that can have strong effects on improving education outcomes, including 
indicators of teaching quality, curriculum content, resources, and new forms of assessment 
(Murnane and Raizen, 1988). Oakes recommended a multi-tiered system of education 




indicators comprised of naiicnal-level indicators, state-level indicators from data collected 
by stales and local districts, and a subset of indicators from both levels that would provide 
a basis for valid state-to-state comparisons and national trends (1986). 

Indicators in education, as in other fields, can include different kinds of statistics 
which may be reported at different levels of aggregation. Some indicators are single 
statistics which are representative indices, some are composites of several statistics, and 
some are disaggregated statistics, such as by economic sector or by state education system. 
Indicators can be designed with different levels of complexity' and aggregation according to 
the needs of different audiences. 

The conceptual framework outlined six components of an indicators model for 
science and mathematics at the state and national levels: (a) student outcomes, (b) 
instructional time/participation, (c) curriculum content, (d) teacher qualitj', (e) school 
conditions, and (f) resources (Blank, 1986), A draft paper was reviewed and revised by an 
advisory panel comprised of state education specialists, scientists, mathematicians, education 
researchers, and representatives of NSF and the Department of Education. The panel used 
the framework to outline three to five "ideal" indicators under each of the six components 
of the model "Ideal indicators" are those that would be most helpful for assessing the 
condition of science and mathematics education regardless of the current availability of data. 



2. Obtain Commitment and Cooperation of Leaders. 

The Council's role in leading the indicators project provided an entree to each state. 
A written agreement to participate in the project was formally obtained from each state 
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superintendent or commissioner. The state's chief state school officer selected staff to 
participate in a project network, including representatives from science and math curriculum, 
student assessment, and information systems. The state superintendents and commissioners 
reviewed the plans for indicator selection, development, and reporting through committees, 
and at their annual meeting, Ha\dng top-level commitment to the project from the outset 
eased the work with state staff on subsequent project steps. 

The polic)' and program uses of the education indicators were emphasized with state 
leaders and state education staff to illustrate the benefits of cooperation. From the initial 
stages, the project was explained as a way for states to improve understanding of their 
education systems, both through useful comparisons with other states and through new ideas 
for analyzing state-collected data and national surveys. Hie Council helped state staff 
organize regional workshops on science and math indicators. Uses of proposed indicators 
impolicy analysis, program planning, and problem identification were outlined and discussed 
with the state representatives. Each workshop was led by national experts on indicators and 
state leaders in science and mathematics education. The state network members were 
invited to participate in several national conferences on science and mathematics indicators, 
including a conference on alteniative methods of assessment and another on measures of 
curriculum content. Finally, the detailed work in developing common definitions and data 
categories for reporting state data was completed with teams of state staff, in order to have 
the system designed to maximize usefulness for states. 

3, Involve Policymakers, Educators, Researchers, and Data Managers in Selecting 
Priori t}' Indicators. 

8 H 



The process of seleciing education indicators requires interaction and consensus 
among different kinds of experts. The selection process should bring together the different 
interests in indicators as well as the expertise needed to organize an indicators sj'Stem based 
on reliable, valid data. Education policymakers and educators have interests in assessin'^ 
the effects of policies and programs and identifying the nature of problems in the education 
system. Researchers are needed to identify variables that are central and critical for 
determining how the system is operating and to pinpoint sources of data that will provide 
valid indicators. Finally, data managers need to participate to ensure that data will be 
available for selected indicators and that the data can be aggregated and reported to meet 
the needs for indicators. 

The process of selecting and developing indicators in other fields has involved 
interaction and consensus building. In analyzing the history of economic indicators, Horn 
and Winter (1989) found that economic indicators were developed during an extensive 
period of interaction among economists, policymakers, and staff of government agencies, 
during which multiple sources of data with different levels of aggregation and complexity 
became important. The process of development and interaction proceeded both prior to 
and after the initiation of data collection and indicators reporting. 

The Council's efforts toward selecting indicators were begun by identifying possible 
sources of data and determining the various needs and interests in state and national 
indicators. All existing national sun'eys that produce state-level data were rcviev/ed. A 
survey was conducted with all 50 state departments of education to identify sources of state 
data and state needs and interests in indicators (CCSSO, 1988). The state survey asked 



about the availability of state data on each of the "ideal indicators," and the survey was 
connpleted by a team of representatives from each State Departnaent of Education. 

The survey responses from states and the information on national surveys were 
tabulated by ideal indicator. Data on many of the ideal indicators were available in only 
a portion of states. For a few indicators, data were available from every state (such as 
.number of science and mathematics teachers). For others, no comparable data were 
available from states but data from a national survey could provide data by state (such as 
student achievement data from the National Assessment of Educational Progress (NCES, 
1991)). A consensus process was needed to weigh the need and importance of proposed 
indicators against data availability and effort required to produce state-by-state indicators, 

A task force was convened comprised of researchers on education indicators, state 
education specialists, state data managers, and federal officials. The group analyzed the 
available data by indicator, and the various interested participants presented their views on 
which indicators should be given highest prio^it)^ Three criteria were used in evaluating and 
prioritizing indicators: 

a. Importance /usefulness of the indicator, 

b. Technical qualir^^ of the data (available or expected), 

c. Feasibility of obtaining state-by-state data, 

Al] three criteria were used to evaluate each possible indicator, but the criterion of 
"imponance/usefulness" was considered first Priority on this criterion is consistent with the 
definition of indicator as having policy or program relevance for the education system. The 
consensus list of priorit}' indicators for science and mathematics included indicators that had 
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very high ratings on importance/usefulness but lower ratings on "feasibility" because data 
are not presently available. Tnis principle is important. For example, in 1985 when the 
Council members approved the initial recommendations for state-by-state indicators, 
comparable data were not available by state for several indicators, including student 
achievement. At the time, many states had achievement tests in the same subjects and 
grade levels, but the variety of items used on the tests prevented valid state-to-siate 
comparisons. However, the decision by the chief state school officers to work toward a 
state-by-siate indicator of student achievement became a major factor supporting expansion 
of the National Assessment of Educational Progress (NAEP) to the state level, which began 
in 1990. Even though the development process was lengthy, it was critical that the chief 
state school officers agreed upon the importance of an indicator of student achievement. 

4. Select a Limited Number of Indicators and Hold Dohti Complexity' in Reporting. 

The task force recommended 12 priority state science/math indicators across sb: 
components of the indicator model, and the indicators are shovm in Figure 1, The group 
decided that, if possible, each of the model components should have at least one priority 
indicator. At the same time, the task force wanted to hold down the total number of 
indicators. This strategy would hold the Ieng:th of reports on the indicators and it would 
focus resources on a small number of critical indicators that can realistically be developed 
at an early point in the process. For the Council's project, it was important to have a 
combination of priority indicators that included some which could be reported uath existing 
state data. This approach allowed the users and potential users of indicators to see the 
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results of indicators work by their states and the benefits of a state and national indicators 
systenri. 

The recommendations for priority indicators also included limiting the amount of 

detail to be reported on any one indicator.^ Indicators should be reported in a format that 

is straight-forward and understandable by different audiences, including educators and 

polic\Tnakers, and the indicators should provide representative or summary statistics about 

the condition of schooling. The indicators may stimulate further analyses and interpretation 

for different specific purposes. Tne role of indicators was well stated by a committee of 

educators in one state: 

Indicators can pro\ade valuable information to guide the debate and dialogue about 
schooling. But judgements about the health of the educational system can only be 
made by interpreting indicator data in the context of educational values and 
experience with schooling. When we track changes [through indicators] its does not 
imply that we have established cause and effect because there are multiple causes. 
Tracking changes may stimulate an hypothesis about the cause and identify areas for 
further investigation... (Massachusetts Association of School Superintendents, 1991, 
p. 23). 

Or ganizing a Cooperative Data System 
5. Decide Method of Collecting Data. 

A critical juncture in development of an indicators system is moving from consensus 
on a desired set of indicators to specifying the measures or data sources for reporting the 
indicators. There are two methodological options for collecting data and reporting statistical 
indicators that are comparable from state to state: 

a. Use the same data collection instrument in all states, or 

b. Implement common standards for categorizing and reporting data, with each 
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siaie using its own data collection instaimeni. 
Use of the same data collection instrument, which applies the same items and 
wording, is essential whenever the information being collected is of a subjective nature and 
is likely to be influenced by the wording or context of the questions themselves. Examples 
of such indicators are student knowledge or skills, student and teacher attitudes, 
instructional methods, school organization, and school processes (Porter, 1991; Oakes, 1989). 

The second option is to apply common standards for data reporting to the different 
organizations or agencies that collect data. Many of our national education, health, and 
labor statistics are based on state and federal cooperation in data collection. The NCES 
Common Core of Data on elementary and secondary education is comprised of a universe 
of data on students, teachers, and schools that are collected by states and districts through 
their management information systems. These state and district data are sometimes called 
"administrative records." Comparability across states is attained not through a sun'ey 
instrument, but rather through common definitions of variables, or data elements, and 
common categories for aggregating and reporting the data. 

One of the priority science and math indicators from state education information 
systems was secondary course enrollments. At the start of the Council project, less than 35 
states collected these data. However, the project moved ahead with data collection and 
reporting with these states, and project staff worked with the remaining states to assist in 
adapting their information systems to obtain course enrollment data. Step 6 describes how 
a cooperative state data system was organized to produce an indicator of course enrollments 
in secondary science and math. 
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6. Work Hith Data Users and Providers to Establish Standards for Producing 
Comparable Data. 

State representatives worked with project staif to set common standards for 
aggregating and reporting state-collected data on course enrollments in science and 
mathematics. There were two main dimensions to the process for setting data reporting 
standards: 

a. To establish common course categories and definitions that would 
incorporate, or provide a link to, most states existing data categories; and 

b. To ensure that the common categories would provide statistics that are useful 
for policymakers and educators. 

For example, it was found that states were very interested in comparing rates of enrollment 

in advanced courses. The project staff and state representatives had to arrive at a definition 

of "advanced" that would allow states to aggregate and report their different courses under 

a common category. 

The process involved extensive cooperation among state managers of information 
systems (data providers) and ,v ;ate science and mathematics specialists (data users). First, 
the project staff determined the extent of variation in course enrollment data by collecting 
and reviewing all of the state data collection forms. The collected forms were analyzed to 
determine state differences in: (a) course categories and definitions; (b) level of intended 
respondent-teacher, school, district; and, (c) other data collected, e.g., student o^nder and 
race/ethnicity, grade level, course difficulty. Tabulations were completed to deternnine the 
extent of commonality and variation across states. For example, the number of states 
collecting data on high school biolog>' was determined, as well as the number using the same 
course categories or codes, e.g., "general biology/' ''basic biology," "college biology," or 
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"honors biolog\'/' 

A planning meeting was held udth state representatives during which several key 
decisions were made. For state and national indicators, the group felt it was most important 
to report and compare enroUment across common "levels" of science and math courses, not 
enrolhnent by course title (or textbook). For example, in science, courses in major subjects 
of biolog)^ chemistr)', and physics should be reported at four levels per subject: first year 
basic, first year general, second year advanced, and second year advanced placement. 

The group also decided that categories should be forward-looking to anticipate the 
data needs of policjTOakers and educators regarding possible changes in science and 
mathematics curriculum. For example, mathematics curriculum is undergoing reform in 
many states, and thus, the high school mathematics categories were created with levels that 
provide tracking of course enrollment in traditional courses (e.g., algebra, geometr>s 
trigonometry), as well as enrolbnent in newer "integrated mathematics" courses. A third 
decision was that the common data categories for state indicators should be sufficiently 
broad to maximize the possibility of matching each state's data to common categories, while 
ensuring comparability and policy relevance. The group also decided that the data system 
should not be modeled after the state with the most detailed data categories nor the one 
with the fewest categories and most simple definitions. 

Reporting Comparative Data on Indicators 

7. Design Data Forms and Cross-Walk Procedure. 
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The Council staff used the s^.ate representatives' decisions and recommendations to 
develop a data reporting form that would give states a common set of instructions and 
specifications for aggregating and reporting their data on science and mathematics. The 
form included a taxonomy of categories for course enrollments with tj'pical course titles 
under each category and written definitions for each categor)'. The definitions were written 
by tearrxs of state specialists in science and mathematics. A draft of the course category 
taxonomy and definitions was circulated to all the state participants in the project network 
for review and comment. Finally, the data reporting design, instructions and forms, and 
cross-walk procedure were tested through a pilot study with eight states. The pilot study 
results were used to refine and clarify the data forms and instructions (CCSSO, 1989). 

For each of the participating states, Council project staff completed a "cross-walk," 
or matching of the common categories, with the state's course codes. For example, a cross- 
walk for one state's data codes with the reporting categories for biology was as follows: 

Common Categories (A State'sl C ourses/Codes 

Biology, 1st Year, Basic/Applied Basic Biology 

Biology, 1st Year General Biologj-; College Preparatory Biology 

Biologj', 2nd Year, Advanced Genetics; Anatomj'; Anatomy and Physiology 

Biology, 2nd Year, Advanced Placement Advanced Placement Biology 
Biolog)', 2nd Year, Other Botany, Ecologv-, Marine Biologj-, Zoology 

8. Collect and Edit Data 

Council staff are responsible for central management of all aspects of the data 
collection from states, including qualit)' control, data editing, and follow-up to obtain timely 
responses from states. Federal funding through the National Science Foundation has made 

16 IS 



possible the central support of the indicators system at the Council. A state contact person 
is responsible for aggregating data, validating the cross-walk procedure, reporting data, and 
verifvdng and correcting the data upon return from the Council data coordinator. 

In the first year of the science and math indicators system (1989-90), data on course 
enrollments were reported by 36 states. The Council assisted states to increase their data 
collection capacity to provide indicators data, and Council staff edited state data to increase 
the quality and accuracy of data. In successive rounds of data reporting through the 
cooperative system, more states will report data as decision-makers see the benefits of 
having the state indicators. Council staff also provide technical assistance and small grants 
of financial support for indicators development through the federal National Science 
Foundation funds. 



9. Report State-by-State Indicators: Example of Course Enrollments in Science and Math 
The first state-by-state report on science and math indicators, including course 
enrollments, comparative state statistics and national totals (Blank and Dalkilic, 1990). The 
first report emphasized reporting individual indicators and establishing baseline numbers for 
later trends analysis. States were encouraged to compare their numbers on a variety of 
indicators with states in their region, states of similar size, and udth national averages. Tlbe 
state network and advisor^' panel did not favor combining indicators into a total "score" for 
each state, and the ranking of states according to indicators was not emphasized. The 
format and design for reporting the indicators reflected important input from state 
representatives and project adWsers. Some examples of statistics from the first report 
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iilusiraie the rationale and key decisions about reporting the state indicators. 

0 Table 1 shows the proportion of public high school students, by state, that 
were estimated to take mathematics ?t three levels by the time they graduate. 
Nationally, 81 percent of students too;, a course at Formal Math Level 1 (e.g., 
alsebra 1), 49 percent took a course at Formal Math Level 3 (e.g., algebra 2), 
and 9 percent took a course at Level 5 (e.g., calculus). 

0 Table 2 shows the proportion of public high school students, by state, that 
were estimated to take science at three leveL by the time they graduate. 
Nationally, over 95 percent of students took a first-year Biology course, 45 
percent took first-year Chemj.str)', and 20 percent took first-year Physics. 

The data were reported by state according to course levels, not course title. This 
provided a way to compare course-taking across states. It also reduced the amount of detail 
that needed to be reported to have a summary indicator of student participation in high 
school science and mathematics. The course-taking rates by state provide comparisons of 
how far students are progressing through the high school curriculum, but they do not provide 
comparisons of course content. 

The selection of three levels for inclusion in each table was based on an interest in 
key gatekeeper courses for science and mathematics. For example, in mathematics, Formal 
Math Level 1 (usually algebra 1) is a gatekeeper for students who wish to complete a formal 
mathematics sequence in high school; Formal Math Level 3 (usually algebra 2) indicates the 
proportion of students who take three years of "high school mathematics;" and, Formal Math 
Level 5 (usually calculus) indicates the proportion of students preparing for science, 
mathematics, or engineering majors in college. 

The rates of course enrollment by state are expressed as the "estimated proportion 
of students taking (a course) by graduation." The percentages are estimates based on the 
actual state enrollments for students in grades 9-12 during one year.' The estimates of 
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course-taking over a four-year high school career were computed because responses to draft 
reports indicated that most readers are used to anal>'zing course-taking rates over a four 
year period. National studies using high school transcripts used this statistic (Kolstad and 
Thome, 1989; Educational Testing Service, 1989), and the state data could be related to the 
national averages from earlier periods of time. 

The national averages for course enrollments shown in Tables 1 and 2 were 
computed with imputation for missing states. The project is working with the remaining 14 
states to obtain course enrollment data, and the succeeding report for 1992 will have more 
states reporting on this indicator.^ The cooperative state data system for science and math 
indicators is a voluntary system. States pay all of the costs of collecting, aggregating editing, 
and reporting the data to the Council, and the central management of the system is from 
federal support. One strategy of the project is to encourage full participation by beginning 
to report state-by-state indicators and to demonstrate how these data can be useful to states. 
State involvement should be a decision based on the needs and interests of each state, not 
because of a national-level requirement or expectation. 

Early dissemination of draft reports was important for producing a final report that 
was useful to policymakers and educators. State participants and project advisers were given 
several opportunities to check the accuracy of the data and to review how the results were 
presented in tables and graphs. Through this process, feedback wets received on organizing 
and formatting state-by-state statistics to improve their usefulness. For example, state 
science and math supervisors prefer to have comparative statistics presented in bar graphs 
which can be used as transparencies for presentations to groups of educators and decision- 
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makers. Suggestions were also made about highlighting certain analyses of course 
enrollment, such as rates of enrollment according to changes in state graduation 
requirements. Some of the uses of the initial report on science and math indicators by 
states include anal)'2ing student course taking patterns, identifying teacher shortages in 
specific fields and focusing teacher training programs, and planning curriculum changes. 

Tne nine steps in developing education indicators are summarized in Figure 2. 
Even though the development process described in these steps was for a state-based system, 
these steps can be used to plan to indicators systems at local, state, and national levels. 



UNRESOLVED ISSUES WITH STATE EDUCATION INDICATORS 

The results from developing a system of state science and math indicators has shown 
that a cooperative state system can produce comparable, useful indicators. The experience 
with science-math indicators has also shown that some specific questions related to 
indicators based on data from state systems are not totally resolved. Three of these issues 
are: (a) At what levels should the indicators data be aggregated and reported? (b) How can 
comparability be assured with data collected by different agencies with different collection 
instruments and var}dng data definitions? (c) How can quality control be maintained v^th 
a cooperative data system? 

The science-math indicators system Vrdth states did design a plan and reporting system 
thai addressed these questions, but more work is yet to be done on each issue. On the first 
issue of level of reporting, a decision was made early in the development of state science- 
math indicators to focus on collecting and reporting state and national totals. However, 




many poliaTOakers and educaiors argue that further disaggregation is needed to district and 
school levels. State averages can mask large variations in the condition and quality of 
education. Also, indicators are typically reported as univariate statistics, such as a state 
average, but important differences require cross-tabulations, such as average class size by 
course level. The Council has begun to do further analysis of the state indicators after the 
initial reporting, and these results v,dil be reported. Workshops are also being held with 
state education agency staff to demonstrate how state data files can be analyzed to produce 
district and school-level indicators. 

One strategy for addressing the issue of comparability with the science-math 
indicators was to keep the indicators categories relatively broad and inclusive. For example, 
with the course taking indicator, it was decided the primary use would be for assessing major 
differences aniong states in course taking levels, not to assess differences in curriculum 
content The indicator was developed through comparing state definitions and arriving at 
a common definition through a consensus process. Each state's courses are placed in 
categories through review of the title and brief descriptions. This process produces 
considerable variation in the courses and airriculum classified within each category. The 
degree of variation in classification needs to be analj'zed through validation research studies, 
and some research is being done on coursework indicators through suppon of the National 
Science Foundation (Burstein, et al, 1993; Smithson, 1993: Stecher, 1992). 

Quality control in the cooperative state system relies heavily on the states. The 
Council staff do central data checking and editing with state totals, and check the response 
rales of the states. This quaUty control helps to improve comparability between states, but 
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the state numbers still rely on the accurac)' of reporting by teachers, schools, and districts, 
as well as data checks and editing by states. Most states conduct computer ediis for data 
reliability and reasonableness. A major question is the accuracy with which rates of course 
enrollment, for example, are placed in the appropriate categories. Another question 
concerns the use- of state data files on teachers. For example, some states lack resources 
to maintain accurate, up-to-date records on current teachers. At the aggregate level, the 
Council staff check state totals with other knowoi data sources, such as at NCES, but cross« 
tabular analyses of teachers from state files do reveal missing and incomplete data. In 
reporting state indicators on science-math teachers, the Council has refrained from 
publishing state numbers that were biased by missing data. Further work is needed by states 
in auditing data reporting and cleaning and updating data files. 

CONCLUSIONS 

At all levels of our education system the topic of education indicators has taken on 
increased attention and significance. Policymakers, educators, and the public want better 
information about the quality of education in our schools. As efforts to collect and report 
data on a set of education indicators have moved forward, the demand has increased for 
more indicators and indicators with greater validity and usefulness. The push to develop a 
system of education indicators at national and state levels that meet polic>TOakers' interests 
in national goals and educational accountability has stimulated educators to ask for 
indicators that are more directly useful in assisting the process of educational improvement. 

This paper has outlined the process and results of developing a set of education 
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indicators system developed through a cooperative data system with states. The experience 
of developing a state education indicators system revealed that their are a variety of issues 
that need to be resolved, and some lessons were gained for others developing indicators. 
Several broad issues have been addressed that apply to any indicators system: (1) Who 
should be involved in selecting the indicators and how can a consensus process be used to 
select and prioritize indicators? (2) How can a cooperative data system be organized to 
produced education indicators? and (3) How should indicators be reported to meet needs 
of users? 

The state science and math indicators were selected through a consensus process that 
involved several interest groups of state policymakers, educators, data managers, and 
researchers. The indicators were designed L^om a research-based model but also carefully 
considered the needs of users of indicators. A consensus approach to selecting education 
indicators is consistent with the position that both rationality and panicipatory democracy 
must have a part in the formulation of an indicators system (Ruby, 1991). Representatives 
of slate policymakers and educators, researchers in science and math education, and state 
and national data managers had a part in determining the indicators that would be given 
priority, as weD as in their design, data coDection, and reporting. 

There are three important steps in selecting indicators by a consensus process. First, 
the broad areas or categories of indicators need to be defined. Second, the kinds of data 
required for the desired indicators need to be identified, and an analysis must be made to 
see if the data are available, valid, and reUable. Then, the critical third step in the process 
is 10 involve the different interests or perspectives on the indicators system in moving from 
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desired or potential indicators to selecting a smaller set of priority indicators for 
implementation. With the state science-math indicators, the state representatives pro\oded 
practical judgements of which indicators would be most informative and useful for 
polic>inakers and educators. The data mangers provided input on existing and planned data 
systems and surveys that would provide dat^; ^ / indicators. The researchers anah'zed 
potential indicators in terms of their reiiabiliLy, validit)', and significance for measuring 
improvement in education. • 

Any system of education indicators is likely to be built from different kinds of data 
sources. Some indicators can be taken directly from large, multi-purpose data bases v^th 
multiple purposes, such as from the U.S. Census. Other indicators can be obtained from 
large-scale sample surveys that have sufficient size to provide disaggregation to different 
levels of the education system, such as from NAEP. The state science and math indicators 
described here provide an example of indicators that rely on existing state data systems and 
the commitment and participation of state staff and resources in implementing the 
indicators. For many states, the use of existing state data was important to show that the 
large amoimt of resources expended to maintain the systems could have uses beyond the 
immediate monitoring and reporting functions within states. In the Council project, the 
development of conunon reporting definitions and categories and a national cooperative 
data system produced a way to use state administrative records for comparative education 
indicators, 

A critical step in developing a cooperative data system was agreeing on common 
standards for aggregating and reponing data that are collected by different agencies. The 
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data caiecories may have lo incorporate different kinds of concerns. For example, the 
categories for course enrollments on science and math had to be both "practical/' to provide 
a way to compare data between existing state data files, and "meaningful," to portray the 
data in terms that give important inform^ation on the status of science and math education. 

The users of indicators were a key consideration in the design of the state science 
and math indicators. They were designed for use primarily at state and national levels. For 
example, a decision was made initially to aggregate and report onJy state-level and national 
statistics, but to also assist states in developing their own indicators that could be reported 
at district or school levels. The state indicators were reported according to data categories 
that would be meaningful to state polic\TOakers and educators, such as advanced vs. 
introductory' levels of course enrollments. The initial report on science-math indicators 
emphasized disaggregation of data for basic analyses of equit)^ such as course enrollments 
by gender and teacher supply and qualifications by gender and race/ethnicity. An iterative 
process was used to give indicators users, data providers, and other advisers an opportunity 
to review the indicators being selected, the data to be reported, and the formats and 
categories in which indicators would be reported. These steps provided a mechanism for 
sharpening the organization and display of indicators to meet the interests and needs of 
primary^ audiences. 



25 

?8 

ERLC 



Notes 

1. This recommendation concerned reporting of the indicators, not collection of data. More 
detailed data related to characteristics of the main indicator, such as student, teacher, or 
school demographics, can be available for use in further analyses. 

2. Each state proportion is a statistical estimate of the course-taking of high school students 
by the time th^.y graduate based on the total course enrollment in grades 9-12 in Fall 1989 
divided by the estimated number of students in a grade cohort during four years of high 
school. The statistical estimating method is imprecise above the 95 percent course-taking 
rate. See Blank and Dalkilic (1990) for further explanation. 

3. A total of 47 states reported state data in 1989-90 for one or more of the indicators in 
the cooperative system, which also included data on science and math teacher characteristics 
and qualifications. 
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Figure 1 

Priority State Indicators of Science and Mathematics Education* 
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DATA SOURCE 


Student Outcomes 
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NAEP 
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NAEP 
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STATE DATA (CCSSO) . 


■.■■■i-ELEMEKTARY ^MINUTES PER WEEK-: .:. ■ ■ 


SCHOOLS/STAFnNG.SURVEY (SASS) ^ 

(NCES) 

: • ■ ■ • •*: • •■• •• J • ■ >•<>: ; :- •. ■ ■ • • • •:• — 
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''Difi noi available .. . '^-'^i^^J^K^'^:^^ 


• - ^.CLASS SIZEby Subjcct/Coonc . . - .V' 
■-. ^vV;V-. N0.,of. C0URSE.PREPARATIONS PER TEACHER, 


StAlB DATA (Available in fnme staici:) • • 


, V;.COURSE OFFERINGS PER* SCHOOL 'jp'' f 'Jy^f ; 
- Teacher Oualiry ' " - ' ^ 




■ ■ COURSES/CREDFTS IN SQENCE/^ : 


SASS (NCES) • 


■ ■: V-.vTEACHING ASSIGNMENTS BY FIELD/SUBJECT ■■■ 


STATE DATA (CCSSO) 


:■ ^-.iTEACHlNG ASSIGNMENTS BY ■ 
■ CERTTFICATIONFTELD/SUBJECr-- ■ 
(Nuir'w of Tcacbcfi Out-of-Ficld). :. 


* ST/ TE DATA (CCSSO) 


• • SsMITC = ■ ' * 




. . i' - . GENDER AND RACE/ETHNICITY ' : 
. .: --=.by Student or Teacher ; ■ . 


STATE DATA (CCSSO) 


: *Approved by CCSSO, November 1987. > 
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Figure 2 

STEPS IN DEVELOPING AN INDICATORS SYSTEM 



Selecting Indicators 

1. Develop a Conceptual Framework Based on Research Results and Interests of 

Policymakers and Educators. 

2. Obtain Commitment and Cooperation of Leaders* 

3- Involve Policymakers, Educators, Researchers, and Data Managers in Selecting Priority 
Indicators. 

4. Select a Limited Number of Indicators and Hold Down Complexity in Reporting* 

Ortyanizing a Cooperative Data Svytem 

5. Decide Method of Collecting Data^ 

6. Work wth Data Users and Providers to Establish Standards for Producing Comparable 

Data. 

Reporting Comparative Data on Indicators 

7. Design Data Forms and Cross-Walk Procedure. 

8. Collect and Edit Data* 

9. Report Indicators. 
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Table 1 

ESTIMATED PROPORTION OF PUBLIC SCHOOL STUDENTS TAKING SELECTED MATHEMATICS 

COURSES BY GRADUATION 


STATE 


ALGEBRA 1 
(FofinaJ Maih Level I) 


ALGEBRA 2 
(FoimaJ Maih L^vd 3) 


CALCULUS 
(FocmiJ Miih Level 5) 


ALABAMA 




46% 


6% 


ALASKA 


— 




— 


ARIZONA 




— 


— 


ARKANSAS 


88 


48 


5 


CALIFORMA 


92 


44 


9 


COLORADO 


— 


» 




coN>rccncuT 


74 


61 


14 


DELAWARE 


73 


43 


17 


DC 


65 


39 


3 


FLORIDA 


78 


42 


9 


GEORGIA 


_ 


— 




HAWAH 


52 


33 


4 


IDAHO 


95+ 


64 


6 


ELLIN OIS 


77 


39 


9 


INDIANA 


60 


45 


8 


lOVr'A 


92 


50 


9 


KANSAS 


66 


47 


9 


KENTUCKY 


SI 


54 


6 


LOUISIANA 


95+ 


64 


4 


MAINE 


S4 


64 




MARYLANT) 


94 


51 


13 


MASSACHUSETTS 


— 




— 


MICHIGAN' 


— 


— 


— 


MINN'ESOTA 


90 


55 


12 


MISSISSIPPI 


85 


58 


3 


MISSOURI 


95 


58 


8 


MONTANA 


94 


65 


6 


NEBRASKA 


75 


54 


6 


NEVADA 


90 


32 


5 


NTW HAMPSHIRE 


■~ 


— 


— 


N^ JERSEY 


— . 


— 


— 


N'EW MEXICO 


95+ 


47 


8 


N'EW YORK 


69 


46 


12 


NORTH CAROLINA 


67 


5i 


8 


NORTH DAKOTA 


95 


64 


3 


OHIO 


80 


47 


8 


OKLAHO^/tA 


95+ 


60 


8 










PENTs'SYLVANU 


88 


57 


16 


RHODE ISLAND 








SOUTH CAROLINA 


69 


55 


7 


SOUTH DAKOTA 








TCNN'ESSEE 


79 


54 


4 


TEXAS 


82 


54 


5 


UTAH . 


82 


63 


13 


VERMONT 








VIRGIMA 


81 


55 


1) 


WASHINGTON 








WESTVIRGINTA 


73 


42 


2 


WISCONSIN 


79 


36 


9 


WYOMING 


73 


29 


8 


U^. TOTAL 


81% 


49% 


9% 



Nou; Each sut£ proportion is i suuistic&l estuniie of count uicuig of high ichool itudenu by the time they sraduaie based on the loul course enrollment 
in jndcs 9-12 in FUl 1989 (See Appendix Table A-5) divided by ihe estimaied number of itudcnu in a jradc oohon during fcxir ycart of high 
school The luoiQcaii estimadng method is imprecise above 95 perccm couree tiionc rat£. (sec Appendix C for further explaniticn) 

Aif eba 1 peroenuges include grade t. 

— Dau no( availible 

U^. Toui«Profx)rtion of tU high school stikdenu estimated to tAke each coune« including impuiAiion fornon-rrponing suies. 
Source: Sate Depurtmcnu of Educaoon. Dau on Public School* FtU 1989; N. Carolina «id Wisconsin. FtU 1988 
Council of Chief Sute School Officert. Suie Education Assessment Center* Washington, DC« 1 990 



Table 2 

ESTIMATED PROPORTION OF PUBLIC HIGH SCHOOL STUDENTS TAKING SELECTED SCIENCE 

COURSES BY GRADUATION 



STATE 



ALABAMA 

ALASKA 

ARIZONA 

ARKANSAS 

CALEFOR>aA 

COLORADO 
CONNBCnCUT 
DELAWARE 
DC 

FLORIDA 

GEORGIA 

HAWAH 

IDAHO 

a.LINOIS 

INDIANA 

IOWA 

KANSAS 

KENTUCKY 

L0U1SL\NA 

MAINE 

MARYLAND 

MASSACHUSETTS 

MICHIGAN 

MINNESOTA 

MISSISSIPPI 

MISSOURI 
MONTANA 
NEBRASKA 
NEVADA 

NEW HAMPSHIRE 

NEW JERSEY 
NEW MEXICO 

YORK 
NORTH CAROLINA 
NORTH DAKOTA 

OHIO 

OKLAHOMA 
OREGON 
PENNSYLVANIA 
RHODE ISLAND 

SOUTH CAROLINA 

SOUTH DAKOTA 

TENNESSEE 

TEXAS 

UTAH 

VERMONT 

vmCENU 

WASHINGTON 

WEST VIRGINU 

WISCONSIN 

WYOMING 



BIOLOGY 
1 Jt Year 



95+% 



95+ 
91 

95+ 
95+ 
75 
95+ 

88 
80 
78 
95+ 

95+ 

95+ 

95+ 

90 

94 

95+ 



95+ 
95+ 

86 
95+ 
95+ 
65 



95+ 
95+ 
95+ 
95+ 

95+ 
93 

95+ 

95+ 

88 

95+ 

80 

95+ 

95+ 
95+ 

86 



CHEMISTRY 
Isi Year 



3$% 



33 
33 

62 
48 
46 
44 

40 
76 
40 
42 

57 
45 
45 

50 
58 

61 



44 

55 

41 
48 
46 
33 



33 
56 
47 
54 

49 
37 

56 
51 

42 
40 
37 

57 

40 

51 
36 



PHYSICS 
Isi Yc«r 



21% 



13 
16 

36 
19 
13 
19 

21 
15 
20 
19 

27 
17 
14 
21 

27 



23 
17 

16 
24 
21 
13 



15 
28 
15 
24 

20 

to 

29 

16 

It 
12 
20 

23 

11 
25 

16 



U.S. TOTAL 



95+% 



45% 



20% 



Note: E*ch lUUi proportion is i iiatisiic*! citimmic of oount ukin^ of high school studcnu by ihe rime they graduiie bistd on ihc louj course airoUmcm 
in gndes 9-12 in FalJ 1 989 (See Appendix Tible A-6) divided by ihe esrijniied number of students in a gndc cx>hoa during four yc*rs of hi gh 
ichooL The luiisricaj esii/nitiiig method is imprecise ibove 95 percent coune taking rate, (jec Appendix C for further explination) 

— Dili not ivtihblc 

U.S. TouJ=:ProporTion of aW high idioo) studcnii esrijnitcd to take each counc, including imputation fornon-reporring suics. 
Source: Siaic Dcpinmcnii of Educadon. Dita on Public Schools. Fall 1 989: N. Caroliju ind Wixconsin. FaU 1988 
Council of Chief Suic School Officcn. Sutr Education Aiscssmcni Ccnicr. Washington. DC, 1990 
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